perf(rknn): price a ladder engine by what the driver reports - #970

What changes

The CosyVoice3 flow-estimator ladder decides which compiled engines it keeps resident with a linear proxy: 5 MiB per compiled frame, against a 2800 MiB ceiling. The proxy exists because the cost of a context is not its image size — all six estimator engines are 661-714 MB on disk and occupy 0.27-3.7 GiB in the NPU I/O aperture — but the proxy is a guess that inherits its shape from the long rungs, and it prices the ladder wrong in both directions:

Two consequences. The cost of a rung is not a function of its compiled length (S=900 is cheaper than S=500), and the proxy's over-estimate on the short rungs makes the ladder refuse sets the board admits: 200+400 costs 3000 MiB by the proxy, so the ladder evicts the short rung, while the board initializes that pair in one process at 3744 MiB. A ladder that could hold two rungs re-inits one of them on every length change — ~2.8 s of rknn_init per request.

How the number is read

engine::rknn::engine_memory_mib(path, label): rknn_init with RKNN_FLAG_COLLECT_MODEL_INFO_ONLY parses the image without building the network (~0.5 s per engine, no NPU memory allocated), and rknn_query(RKNN_QUERY_MEM_SIZE) then reports total_weight_size + total_internal_size — the bytes the real init goes on to allocate, less a per-context slack of 30-50 MiB measured on this board. It returns 0 when the driver cannot say, so each sidecar keeps its own proxy as the fallback. Called once per engine at ladder registration, not per request.

ApertureBudget now counts MiB, its only unit, and keeps no ceiling of its own; the ceiling stays model-local with the matrix it was derived from (kAperturePinBudgetMiB = 3900 in rknn_flow.cpp).

Runtime validation (RK3588, Orange Pi 5 Plus, 8 GiB, librknnrt 2.3.2)

Board build: cmake --build /root/audio.cpp/build -j 8 --target audiocpp_server with -DENGINE_ENABLE_RKNN=ON -DENGINE_RKNN_INCLUDE_DIR=/usr/local/include -DENGINE_RKNN_LIBRARY=/usr/lib/librknnrt.so, model set piper_tts,kokoro_tts,qwen3_tts,cosyvoice3,f5_tts,voxcpm2.

Two binaries from one tree — main (bin-proxy) and this branch (bin-measured) — one server config listing all six rungs
(/root/srvconc/server-rknn-full.json, cosyvoice3.exec_plan → flow_estimator_seq{110,200,250,400,500,900}.rknn
from /root/audio.cpp/models/CosyVoice3-RKNN/), GGUF
/root/audio.cpp/models/CosyVoice3-GGUF/cosyvoice3-q8_0.gguf, runner
/root/pinladder/ab9-run.sh: cold server, six POST /v1/audio/speech
(--backend cpu --threads 4, zero-shot voice from /root/srvconc/ref.wav, seed=1234) alternating
between the S=200 and S=400 rungs (n=4,50,10,50,4,50 chars → 118/308/158/308/118/308 mel frames).

The saving is exactly one rknn_init per request that needs no new engine; the price is the second engine's residency, visible in RSS. Registration logs the table (cosyvoice3.flow.dit.rknn.aperture_mib = 786/1129/1426/2615/3752/1039 on this board) and resident_mib the live set.

Ceiling evidence, same board, sets initialized in one process (/root/memq/pairs.c): 110+200+250 (3341 MiB), 110×4 (3144), 200×3 (3387), 250+250+110 (3638), 110+400 (3401) and 200+400 (3744) all rknn_init cleanly, while 250+400 (4041), 110+500 (4538) and every set holding an S≥900 engine return -4. 3900 MiB is that measured boundary, not a board property — the accounting is shared, the numbers are this engine's.

Trace showing the policy difference:

main:       loaded_frames 200 → resident_frames 200 → loaded_frames 400 → resident_frames 400 →  (6 loads)
this branch: loaded_frames 200 → resident_mib 1129 → loaded_frames 400 → resident_mib 3744 →     (2 loads)

Host

rknn_engine_test passes (ENGINE_BUILD_EXTENDED_TESTS=ON; the suite is registered under that gate): the served ladder trace re-run with the measured costs gives 2 loads and a 2555 MiB resident set, 200+400 fits, 250+400 (4041) and 110+500 (4538) do not, and S=900's own 1039 MiB fits the aperture while the engine still fails at its own IOVA ceiling. python3 tools/check_loader_catalog_sync.py --self-test && python3 tools/check_loader_catalog_sync.py → ok.

Known limits

• The per-engine costs and the 3900 MiB ceiling are measurements of this estimator on this board. A family that wants the shared accounting has to measure its own; the type takes MiB in and holds no ceiling, so nothing here imports a CosyVoice3 number into another sidecar.
• engine_memory_mib costs ~0.5 s per engine at startup (3 s for a six-rung ladder), once, in exchange for the loads it removes per request.
• The S≥900 engines are still refused by their own per-engine ceiling (~4.7 GB requested) and fall back to the CPU graph; the measured model does not change that.